{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "# Federated Learning Training Plan: Host Plan & Model\n",
    "\n",
    "Here we load Plan and Model params created earlier in \"Create Plan\" notebook, host them to PyGrid, \n",
    "and run sample syft.js app that executes them.  "
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "outputs": [],
   "source": [
    "import asyncio\n",
    "import websockets\n",
    "import json\n",
    "import binascii\n",
    "\n",
    "async def sendWsMessage(data):\n",
    "    async with websockets.connect(gatewayWsUrl) as websocket:\n",
    "        await websocket.send(json.dumps(data))\n",
    "        message = await websocket.recv()\n",
    "        return json.loads(message)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n",
     "is_executing": false
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Step 5a: Host in PyGrid\n",
    "\n",
    "**NOTE**: Right now PyGrid supports only 1 version of plan (list of ops or torchscript).\n",
    "But Plans will converge to one type here https://github.com/OpenMined/PySyft/issues/2994#issuecomment-595333791.\n",
    "\n",
    "Here we load \"ops list\" Plan."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "outputs": [],
   "source": [
    "# Load files with protobuf created in \"Create Plan\" notebook.\n",
    "with open(\"model_params.pb\", \"rb\") as model_file, open(\"tp_ops.pb\", \"rb\") as plan_file: \n",
    "    serialized_model = model_file.read()\n",
    "    serialized_ops_plan = plan_file.read()\n"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% \n",
     "is_executing": false
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Follow PyGrid README.md to build `openmined/grid-gateway` image from the latest `dev` branch \n",
    "and spin up PyGrid using `docker-compose up`."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "outputs": [],
   "source": [
    "# Default gateway address when running locally \n",
    "gatewayWsUrl = \"ws://127.0.0.1:5000\""
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n",
     "is_executing": false
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "This hand-crafted request emulates `pygrid.host_federated_training` from the roadmap."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "outputs": [],
   "source": [
    "# These name/version you use in worker\n",
    "name = \"mnist\"\n",
    "version = \"1.0.0\"\n",
    "\n",
    "host_request = {\n",
    "    \"type\": \"federated/host-training\",\n",
    "    \"data\": {\n",
    "        \"model\": binascii.hexlify(serialized_model).decode(\"utf-8\"),\n",
    "        \"plans\": {\n",
    "            \"training_plan\": binascii.hexlify(serialized_ops_plan).decode(\"utf-8\"),\n",
    "            # \"extra_plan\": ...,\n",
    "        },\n",
    "        \"protocols\": {\n",
    "            # \"secure_agg_protocol\": ...,\n",
    "        },\n",
    "        \"averaging_plan\": \"\",\n",
    "        \"client_config\": {\n",
    "            \"name\": name,  \n",
    "            \"version\": version,\n",
    "            \"batch_size\": 64,\n",
    "            \"lr\": 0.01,\n",
    "            \"max_updates\": 200  # custom syft.js option that limits number of training loops per worker\n",
    "        },\n",
    "        \"server_config\": {\n",
    "            \"max_workers\": 100,\n",
    "            \"pool_selection\": \"random\",\n",
    "            \"num_cycles\": 5,\n",
    "            \"do_not_reuse_workers_until_cycle\": 4,\n",
    "            \"cycle_length\": 28800,\n",
    "            \"minimum_upload_speed\": 0,\n",
    "            \"minimum_download_speed\": 0\n",
    "        }\n",
    "    }\n",
    "}"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n",
     "is_executing": false
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Shoot!\n",
    "\n",
    "If everything's good, success is returned.\n",
    "If the name/version already exists in PyGrid, change them above or cleanup PyGrid db by re-creating docker containers. \n"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "outputs": [
    {
     "name": "stdout",
     "text": [
      "Host response: {'status': 'success'}\n"
     ],
     "output_type": "stream"
    }
   ],
   "source": [
    "response = await sendWsMessage(host_request)\n",
    "print(\"Host response:\", response)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n",
     "is_executing": false
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Let's double-check that data is loaded by requesting a cycle."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "outputs": [
    {
     "name": "stdout",
     "text": [
      "Auth response:  {\n",
      "  \"status\": \"success\",\n",
      "  \"worker_id\": \"0c785fc4-be45-45e7-b269-82ba6781c425\"\n",
      "}\n",
      "Cycle response: {\n",
      "  \"status\": \"accepted\",\n",
      "  \"request_key\": \"5be5738875c8b2afb6c3a9f351b20cd9760f4d6c2a3b129dcfbcc2c0a6bcfa39\",\n",
      "  \"model\": \"mnist\",\n",
      "  \"plans\": {\n",
      "    \"training_plan\": 2\n",
      "  },\n",
      "  \"protocols\": {},\n",
      "  \"client_config\": {\n",
      "    \"name\": \"mnist\",\n",
      "    \"version\": \"1.0.0\",\n",
      "    \"batch_size\": 64,\n",
      "    \"lr\": 0.01,\n",
      "    \"max_updates\": 200\n",
      "  },\n",
      "  \"model_id\": 1\n",
      "}\n"
     ],
     "output_type": "stream"
    }
   ],
   "source": [
    "auth_request = {\n",
    "    \"type\": \"federated/authenticate\",\n",
    "    \"data\": {}\n",
    "}\n",
    "auth_response = await sendWsMessage(auth_request)\n",
    "print('Auth response: ', json.dumps(auth_response, indent=2))\n",
    "\n",
    "cycle_request = {\n",
    "    \"type\": \"federated/cycle-request\",\n",
    "    \"data\": {\n",
    "        \"worker_id\": auth_response['worker_id'],\n",
    "        \"model\": name,\n",
    "        \"version\": version,\n",
    "        \"ping\": 1,\n",
    "        \"download\": 1000,\n",
    "        \"upload\": 1000,\n",
    "    }\n",
    "}\n",
    "cycle_response = await sendWsMessage(cycle_request)\n",
    "print('Cycle response:', json.dumps(cycle_response, indent=2))\n"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n",
     "is_executing": false
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Step 6a: Train\n",
    "\n",
    "Start and open \"with-grid\" example in syft.js project (http://localhost:8080 by default), \n",
    "enter model name and version and start FL training.\n",
    "\n",
    "\n"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  },
  "pycharm": {
   "stem_cell": {
    "cell_type": "raw",
    "source": [],
    "metadata": {
     "collapsed": false
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}